Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Bivariate Extreme Value Copulas #208

Merged
merged 43 commits into from
Aug 3, 2024
Merged

Conversation

Santymax98
Copy link
Contributor

new file:   docs/src/extremevalue/generalities.md
modified:   src/Copulas.jl
new file:   src/ExtremeValue.jl
new file:   src/ExtremeValueCopulas/AsymGalambosCopula.jl
new file:   src/ExtremeValueCopulas/AsymLogCopula.jl
new file:   src/ExtremeValueCopulas/AsymMixedCopula.jl
new file:   src/ExtremeValueCopulas/BC2Copula.jl
new file:   src/ExtremeValueCopulas/CuadrasAugeCopula.jl
new file:   src/ExtremeValueCopulas/GalambosCopula.jl
new file:   src/ExtremeValueCopulas/HuslerReissCopula.jl
new file:   src/ExtremeValueCopulas/LogCopula.jl
new file:   src/ExtremeValueCopulas/MOCopula.jl
new file:   src/ExtremeValueCopulas/MixedCopula.jl
new file:   src/ExtremeValueCopulas/tEVCopula.jl
new file:   src/UnivariateDistribution/ExtremeDist.jl
new file:   test/Extreme_value_test.jl

	new file:   docs/src/extremevalue/generalities.md
	modified:   src/Copulas.jl
	new file:   src/ExtremeValue.jl
	new file:   src/ExtremeValueCopulas/AsymGalambosCopula.jl
	new file:   src/ExtremeValueCopulas/AsymLogCopula.jl
	new file:   src/ExtremeValueCopulas/AsymMixedCopula.jl
	new file:   src/ExtremeValueCopulas/BC2Copula.jl
	new file:   src/ExtremeValueCopulas/CuadrasAugeCopula.jl
	new file:   src/ExtremeValueCopulas/GalambosCopula.jl
	new file:   src/ExtremeValueCopulas/HuslerReissCopula.jl
	new file:   src/ExtremeValueCopulas/LogCopula.jl
	new file:   src/ExtremeValueCopulas/MOCopula.jl
	new file:   src/ExtremeValueCopulas/MixedCopula.jl
	new file:   src/ExtremeValueCopulas/tEVCopula.jl
	new file:   src/UnivariateDistribution/ExtremeDist.jl
	new file:   test/Extreme_value_test.jl
@Santymax98
Copy link
Contributor Author

Hi dear @lrnv Could you help me find out an error that appears in tEVCopula? Maybe the error is that we don't have StatsFuns as a deps??

tEVCopula - sampling, pdf, cdf: Test Failed at /home/runner/work/Copulas.jl/Copulas.jl/test/Extreme_value_test.jl:293
  Expression: e isa ArgumentError
   Evaluated: UndefVarError(:StatsFuns) isa ArgumentError
Stacktrace:
 [1] macro expansion
   @ ~/work/Copulas.jl/Copulas.jl/test/Extreme_value_test.jl:293 [inlined]
 [2] macro expansion
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
 [3] top-level scope
   @ ~/work/Copulas.jl/Copulas.jl/test/Extreme_value_test.jl:262

The other errors are something related to

test/margins_uniformity.jl | 1251 11 1262

I think they have nothing to do with Extreme Value Copulas

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

I will take a look tonight -- stuck at a airport. The margins_uniformity checks that every copula has uniform margins, and if your copula do not pass the test then yes it's an issue :)

@Santymax98
Copy link
Contributor Author

Santymax98 commented Jul 25, 2024 via email

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

For work, but then a flights got cancelled... So, does hits PR replaces #205 ? If yes, we could close #205 then. Looking at the errors right now will be back soon

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

So, margins_uniformity.jl is simply a script that checks for uniformity of the margins of the copulas. I yells when there are copulas types without instances tested, which was your case. I added some instances of your types in the list.

Now it yells because some of them do not produce marignals uniform enough. We have to investigate why -- maybe the way i'm checking uniformity is not right !

@Santymax98
Copy link
Contributor Author

Yup, this replaces #205 ;

The algorithm I used is the one presented in this reference Ghoudi, K., Khoudraji, A., & Rivest, E. L. P. (1998). Propriétés statistiques des copules de valeurs extrêmes bidimensionnelles. Canadian Journal of Statistics, 26(1), 187-197. in section 3.

Santymax98 and others added 5 commits July 25, 2024 17:20
	modified:   src/Copulas.jl
	modified:   src/ExtremeValueCopulas/tEVCopula.jl
	modified:   src/UnivariateDistribution/ExtremeDist.jl
@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

So I temporarily removed the KS test so that we can concentrate on the rest for the moment.

I have weird behaviors of HuslerReis (and also tEV it seems ?) on the cdf when one of the two values is 1 :

julia> C = HuslerReissCopula(0.1)
HuslerReissCopula{Float64}=0.1)

julia> cdf(C, [1,0.7])
ERROR: DomainError with -Inf:
log was called with a negative real argument but will only return a complex result if called with a complex argument. Try log(Complex(x)).
Stacktrace:
 [1] throw_complex_domainerror(f::Symbol, x::Float64)
   @ Base.Math .\math.jl:33
 [2] _log(x::Float64, base::Val{:ℯ}, func::Symbol)
   @ Base.Math .\special\log.jl:301
 [3] log
   @ .\special\log.jl:267 [inlined]
 [4] (H::HuslerReissCopula{Float64}, t::Vector{Float64})
   @ Copulas c:\Users\lrnv\.julia\dev\Copulas\src\ExtremeValueCopulas\HuslerReissCopula.jl:45
 [5] _cdf(C::HuslerReissCopula{Float64}, u::Vector{Float64})
   @ Copulas c:\Users\lrnv\.julia\dev\Copulas\src\ExtremeValue.jl:25
 [6] cdf(C::HuslerReissCopula{Float64}, u::Vector{Float64})
   @ Copulas c:\Users\lrnv\.julia\dev\Copulas\src\Copula.jl:19
 [7] top-level scope
   @ REPL[83]:1

julia> 

And this is precisely the checks I am doing : checking that cdf(, [1,....1, x, 1.... 1]) == x for any x to ensure margins uniformity.

Maybe you could ammend your implementation to return the correct limit instead of failling in some cases ? maybe there are logs that need to be more carefully implemented or exp(log(..)) that coul dbe removed, or something like that ?

@Santymax98
Copy link
Contributor Author

I'm going to review the edge cases when they are 1 and 0, the way to get the cdf is exp(-\ell)

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

Ok i think i managed to fix some of the domain errors by replacing a t = -log.(u) by t = abs.(log.(u)), where 0 <= u <= 1. This is sementically the same but then -log(1) is -0 while you intended 0.

Runnig the full test suite again to see what's next :)

Santymax98 and others added 2 commits July 25, 2024 18:30
@Santymax98
Copy link
Contributor Author

Using absolute value was a good move

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

Now looks like MOCopula is failling the uniformity test of its marginals:

(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.30184329672330246, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.3835528626511967  0.30184329672330246 (atol=1.0e-5)

Stacktrace:
 [1] macro expansion
   @ C:\Users\lrnv\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [2] top-level scope
   @ C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.6965294668307551, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.7487754473247006  0.6965294668307551 (atol=1.0e-5)

Stacktrace:
 [1] macro expansion
   @ C:\Users\lrnv\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [2] top-level scope
   @ C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.6770337002357543, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.7319614847835554  0.6770337002357543 (atol=1.0e-5)

Stacktrace:
 [1] macro expansion
   @ C:\Users\lrnv\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [2] top-level scope
   @ C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.5793654774234129, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.6461927642785321  0.5793654774234129 (atol=1.0e-5)

Stacktrace:
 [1] macro expansion
   @ C:\Users\lrnv\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [2] top-level scope
   @ C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.6629483465032526, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.7197534469127619  0.6629483465032526 (atol=1.0e-5)

Stacktrace:
 [1] macro expansion
   @ C:\Users\lrnv\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [2] top-level scope
   @ C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.30184329672330246, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.3835528626511967  0.30184329672330246 (atol=1.0e-5)

Stacktrace:
 [1] macro expansion
   @ C:\Users\lrnv\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [2] top-level scope
   @ C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.6965294668307551, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.7487754473247006  0.6965294668307551 (atol=1.0e-5)

Stacktrace:
 [1] macro expansion
   @ C:\Users\lrnv\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [2] top-level scope
   @ C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.6770337002357543, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.7319614847835554  0.6770337002357543 (atol=1.0e-5)

Stacktrace:
 [1] macro expansion
   @ C:\Users\lrnv\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [2] top-level scope
   @ C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.5793654774234129, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.6461927642785321  0.5793654774234129 (atol=1.0e-5)

Stacktrace:
 [1] macro expansion
   @ C:\Users\lrnv\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [2] top-level scope
   @ C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
(C, u) = (Copulas.MOCopula{Float64}(λ1=0.1, λ2=0.2, λ12=0.3), [0.6629483465032526, 1.0])
Generic tests on every copulas: Test Failed at C:\Users\lrnv\.julia\dev\Copulas\test\margins_uniformity.jl:146
  Expression: (cdf(C, u), val, atol = 1.0e-5)
   Evaluated: 0.7197534469127619  0.6629483465032526 (atol=1.0e-5)

Basically you can verify it by runnig:

C = MOCopula(0.1,0.2,0.3)
u = [1, rand()]
v = [rand(), 1]
(cdf(C,u), u[2]) # should be equal
(cdf(C,v), v[1]) # should be equal

Moreover, it looks like it is OK for [1, rand()] but failling for [rand(), 1]. Maybe you need to check the implementation of this cdf ?

@Santymax98
Copy link
Contributor Author

using Distributions
using HypothesisTests
using TwinCopulas

function test_uniformity(data)
    n, m = size(data)
    results = []
    
    for i in 1:n
        sample = data[i, :]
        ks_test = ApproximateOneSampleKSTest(sample, Uniform(0, 1))
        p_value = pvalue(ks_test)
        push!(results, (i, p_value))
    end
    
    return results
end

cops = (AsymGalambosCopula(0.1, [0.2,0.6]),
        AsymLogCopula(1.2, [0.3,0.6]),
        BC2Copula(0.7,0.3),
        CuadrasAugeCopula(0.1),
        HuslerReissCopula(0.1),
        MixedCopula(0.2),
        MOCopula(0.1,0.2,0.3),
        )

for (i, C) in enumerate(cops)
    println("\nProbando copula $i: $(typeof(C))")
    data = rand(C, 1000) # Genera datos a partir de la copula

    results = test_uniformity(data)

    # Mostrar los resultados
    for (marginal, p_value) in results
        println("Marginal $marginal: p-value = $p_value")
        if p_value < 0.05
            println("Rechazamos la hipótesis nula de que la marginal $marginal proviene de una distribución uniforme.")
        else
            println("No podemos rechazar la hipótesis nula de que la marginal $marginal proviene de una distribución uniforme.")
        end
    end    
end

In my case I used this, and it checks uniformity for all, I have problems with Galambos and logistic (which should be equal to gumbel)

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

The problem with this test is that it might just fail out of bad luck (that is the definition of a 5% p-value : in 5% of the cases it'll fail under H0)...

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

You have missing references in the .bib : Joe1990, Ghoudi1998. can you add them ?

@Santymax98
Copy link
Contributor Author

You are right, I also noticed the bug in MOCopula, we could remove the function $\ell$ from the file and leave only $\isansA$

@Santymax98
Copy link
Contributor Author

I'm terrible at placing citations, however I think I can look them up and start completing each one of them. I think I only put the names of the references without links, right, or .bib, right?

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

The abstract type ExtremeValueCopula is not documented, while you make a reference to it in the docs. Maybe it shoul dbe documented ?

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

I'm terrible at placing citations, however I think I can look them up and start completing each one of them. I think I only put the names of the references without links, right, or .bib, right?

Yes you wrte the citations corectly, just forgot to add them to the .bib file

@lrnv
Copy link
Owner

lrnv commented Jul 25, 2024

You are right, I also noticed the bug in MOCopula, we could remove the function ℓ from the file and leave only \isansA

If it is enough, yes you can try that

@Santymax98
Copy link
Contributor Author

Santymax98 commented Jul 29, 2024 via email

@lrnv
Copy link
Owner

lrnv commented Jul 29, 2024

Sorry to hear that, hope you'll get better soon ! :/

Yeah take your time, there is no rush on my side. I also see that there are one or two more copulas in your twincopulas that could me migrated to here. Also statistics such as Blomqvist \beta & others could be migrated too if you want, with potentail restrictions to $d=2$ by the type system directly.

Once all of that is done, I could help you design dynamic stuff on top of the class system that is here if you want / need !

Let's stay in touch.

++
Oskar

@Santymax98
Copy link
Contributor Author

Santymax98 commented Jul 29, 2024 via email

Copy link

codecov bot commented Jul 29, 2024

Codecov Report

Attention: Patch coverage is 81.00264% with 72 lines in your changes missing coverage. Please review.

Project coverage is 86.32%. Comparing base (b9bc7e4) to head (42568e4).
Report is 67 commits behind head on main.

Files Patch % Lines
src/ExtremeValueCopula.jl 59.72% 29 Missing ⚠️
src/ExtremeValueCopulas/tEVCopula.jl 84.37% 15 Missing ⚠️
src/ExtremeValueCopulas/BC2Copula.jl 66.66% 10 Missing ⚠️
src/ExtremeValueCopulas/AsymMixedCopula.jl 77.77% 4 Missing ⚠️
src/UnivariateDistribution/ExtremeDist.jl 84.00% 4 Missing ⚠️
src/ExtremeValueCopulas/AsymGalambosCopula.jl 85.00% 3 Missing ⚠️
src/ExtremeValueCopulas/AsymLogCopula.jl 76.92% 3 Missing ⚠️
src/ExtremeValueCopulas/CuadrasAugeCopula.jl 86.95% 3 Missing ⚠️
src/ExtremeValueCopulas/MOCopula.jl 96.29% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #208      +/-   ##
==========================================
+ Coverage   83.42%   86.32%   +2.90%     
==========================================
  Files          29       44      +15     
  Lines         748     1170     +422     
==========================================
+ Hits          624     1010     +386     
- Misses        124      160      +36     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Santymax98
Copy link
Contributor Author

Santymax98 commented Jul 29, 2024

Hello Oskar, I added the CDF directly, it appears that all the tests passed... would I need to start working with the documentation? In relation to Blomqvist \beta and giny´s coefficient \gamma, maybe we could do it in another PR?

@lrnv
Copy link
Owner

lrnv commented Jul 30, 2024

Yes beta and gamma would go in another PR that makes sense.

On this one you can correct the docs and we:ll be alright.

GG for MOCopula !

@lrnv
Copy link
Owner

lrnv commented Jul 31, 2024

II write it here as i think about it now: you could add up (in this PR) a line in the comparison table on the readme about bivariate extreme value copulas

@Santymax98
Copy link
Contributor Author

Santymax98 commented Jul 31, 2024 via email

	modified:   docs/src/assets/references.bib
	modified:   docs/src/extremevalue/generalities.md
	modified:   src/ExtremeValueCopula.jl
@lrnv
Copy link
Owner

lrnv commented Aug 1, 2024

The erros "Invalid sequence" seems to mean that you did not format the math correctly in the docstring : take a look at other docstrings (e.g. archimedeans) to see how it should be done.

@lrnv
Copy link
Owner

lrnv commented Aug 1, 2024

The output on Julia 1.10 is way better than on julia 1.6, it gives:

ERROR: LoadError: ParseError:
# Error @ /home/runner/work/Copulas.jl/Copulas.jl/src/ExtremeValueCopula.jl:26:118

In this way, in order to define a bivariate copula of extreme values, it is only necessary to introduce the function \( A\).
#                                                                                                                    └┘ ── invalid escape sequence
Stacktrace:

@Santymax98
Copy link
Contributor Author

@lrnv Do you think it's ready? Or do we need to add something else?

@lrnv
Copy link
Owner

lrnv commented Aug 2, 2024

I will review it tomorrow before merging but I think it's ok like that

@lrnv lrnv merged commit f8eb5aa into lrnv:main Aug 3, 2024
7 checks passed
@lrnv
Copy link
Owner

lrnv commented Aug 3, 2024

@Santymax98 see #210 there is an issue in the docs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants